Development of Informal Communication Environment Using Interactive Tiled Display Wall

In the early stage of planning or design, it is desired to support the informal communication between remote sites. In this study, informal communication environment between remote places was constructed by using the interactive tiled display wall. This system measures the user's gesture as well as captures the user's image by using the multiple fish-eye cameras. Then, the remote users can communicate with each other by using the touch interaction in the shared space. This system was applied to the remote communication and the effectiveness of this method in the informal communication was experimentally evaluated. Introduction In the development of large-scale systems or products, it is important to build the environment where collaborative work among researcher, designer, and manager who exist in physically distant places can be performed smoothly. In the latest CAD, CAE and PDM systems, concurrent engineering based on data sharing has been realized to some extent [1]. Moreover, in the video conferencing systems, communication between remote places has been realized by transmitting the users’ live videos through the Internet [2]. However, in the early stage of planning or design, it is desired to support the informal communication so that new ideas may be created easily, and this kind of supporting system has not been fully realized. In this study, informal communication support system between remote places that uses the interactive tiled display wall equipped with the fish-eye cameras was developed. In the following sections, concept of the proposed system, some elementary technologies such as video avatar and pointing interaction, and evaluation experiment are described Concept of Tiled Display Communication In order to realize the informal communication that is used in the early stage of planning or design, it is important to support the natural communication between remote users as if they are talking face to face. As the functions of informal communication support system, several requirements such as full scale expression of user, eye contact between remote users, usage of pointing gesture, and information sharing between remote sites are considered. Namely, it is required that remote users can communicate with each other not on the desktop terminal but in the shared space while sharing necessary information freely. In this study, in order to realize such functions, the communication system using the tiled display wall was used. The tiled display is a super high resolution large display that consists of multiple LCD monitors [3]. In Particular, in this study, tiled display system equipped with fish eye cameras was developed [4]. Figure 1 shows the concept of the proposed informal communication support system. In this system, a high resolution large screen display is constructed by using the tiled display wall, so that the full scale image of the user can be represented. The tiled display wall consists of twelve (4x3) 20inch LCD monitors (Dell 2007FP). Since the resolution of each monitor is 1600x1200, the total resolution of the tiled display wall is 6400x3600. In this display, SAGE library is used to render the image on the tiled multiple monitors [5]. Moreover, in this system, several fish-eye cameras (KEIYO SKP-R705 VHB, Minilens 1.24mm) are attached to the bezels between monitors. Since the fish-eye camera can capture the scene with wide viewing angle of about 180 degrees, it can sense the whole space in front of the tiled display wall. Therefore, the camera image can be used to measure the user's gesture as well as to capture the image of the user who is standing in front of the tiled display. By using these functions, video avatar and touch interaction technology were implemented to realize the effective collaborative work between remote places. Figure 2 shows the tiled display system equipped with the multiple fish-eye cameras that was developed in this study. Figure 1. Concept of informal communication Figure 2. Tiled display system using multiple support system fish-eye cameras Video Avatar Communication Video avatar is a method of realizing high presence communication in the shared virtual world by transmitting the full scale live video image of the users mutually between remote places [6]. In this study, in order to generate a video avatar, fish eye cameras were used. Figure 3 shows the process of generating the video avatar. First, the image of the user who is standing in front of the tiled display is captured by the fish-eye camera, and only the user's figure is segmented from background using background subtraction method. In this method, the background image without users is captured beforehand, and when the user's image is captured, the difference between the captured image with the user and the background image without the user is calculated in each frame. Next, the fish-eye distortion that is contained in the captured wide viewing angle image is corrected. In this process, the camera parameters and the distortion values in the captured image are calculated beforehand from the captured image of the checkerboard pattern using OpenCV library, and they are applied to the captured image of the user to correct the distortion. However, surrounding area of the fish-eye image cannot be corrected completely because the size of the checkerboard image is restricted. Therefore, the portion of 260 by 260 pixels in the center area of the corrected image is cut out and it is sent to the remote site. By using this method, it has become possible to represent the remote user's image overlapped on the displayed information and it can be used for the video avatar communication. Though the fish-eye camera captures the image of the whole space in front of the tiled display wall, the user's figure seen from the slant direction is captured when he is standing diagonally in front of the camera. In this study, experiment on measuring the eye contact in the video avatar communication based on the image captured by the fish-eye camera was conducted. In the experiment, five subjects were asked to answer the questions "Do you feel you are being looked at by the video avatar?", "Do you feel you are being talked to by the video avatar?" and "Do you feel you want to talk to the video avatar?" using five grade system (2: strongly agree, 1: somewhat agree, 0: neither agree nor disagree, -1: somewhat disagree, -2: strongly disagree), when they looked at the video avatar image that was captured by the fish-eye camera from the direction of -5.0, -2.5, 0.0, 2.5, 5.0 degrees respectively. Figure 4 shows the result of the experiment. From the result, when the direction of the camera that captured the user's figure was larger than 5.0 degrees, the user’s feeling of eye contact and talking with the video avatar was decreased. Namely, we can understand when the large tiled display wall remote user sensing space user sensing space fish-eye camera network tiled display wall were used, the ideal conditions of the camera position and the user’s standing position should be considered for the effective communication. Figure 3. Process of generating video avatar. Figure 4. Result of video avatar experiment. Pointing Interaction Since the fish-eye camera has wide viewing angle of about 180 degrees, it can capture the wide image up to monitor surface. Therefore, when the user points at the monitor surface, the fingertip position is recorded in the peripheral area of the fish-eye image, and the direction of the fingertip seen from the camera position can be identified. When the directions of the fingertip are identified by two fisheye cameras, the fingertip position on the tiled display surface can be calculated as the intersection point of two directions. Figure 5 shows the method of measuring the fingertip position of the user based on the images of the fish-eye cameras. Namely, the fingertip position (x, y) on the tiled display surface is calculated from the directions of α and β based on the equation (1), and it can be used for the pointing interaction. ) 1 ( sin cos cos sin sin sin , sin cos cos sin sin cos   a y a x β α β α β α β α β α β α + = + = This method measures not the touched position on the screen surface but the fingertip position approaching the screen. In this case, though the pointing position can be measured without touching the screen surface, the accuracy of the measured position is not so high. In this study, the experiment on measuring the accuracy of touching position on the tiled display surface was conducted. In the experiment, the subjects were asked to touch the positions indicated at 400 pixels intervals on the tiled display surface, and the accuracy of the measured positions was evaluated. As a result, the average errors along the horizontal axis (x) and vertical axis (y) were 144.6 pixels (3.7 cm) and 107.5 pixels (2.7 cm), respectively. As for the positions, there was no significant difference among the touched positions. From the result of this experiment, we can understand that though this method is not suitable for the precise collaborative work, it can adequately be used for the pointing interaction in the informal communication. fish-eye image distortion correction background subtraction video avatar